Statistics for the Performance Analyst

Many engineers come into the performance world from different backgrounds and lack the appropriate education. For example, a software developer may be challenged to look at the performance problems in some code and might transition into software testing as a result. He learns what he needs to know on-the-fly. Many times, he does not know that he needs to know something and does what he can. If there is any review, he learns from his mistakes. Statistics is one of those fields that is necessary for performance analysis, but may not be given proper attention in advance. It might just be that it has been years since you encountered the topic in high school or college. This issue has been addressed in the literature before ([Mac89], [Kal02]). For some, basic statistics is sufficient. For others, more advanced knowledge is necessary. This paper is an introduction to statistics from the performance analyst’s perspective. It will focus on understanding the common concepts in order to use them rather than implement them. Many tools (e.g., Excel, R) implement the concepts and the analyst just needs to understand how to use them. So, we will not be writing out equations nor deriving them. For more details in any particular area, numerous materials exist ([Sto98], [NIS06], [Lan07], and [wik10, “Statistics”]). Strangely, only a few performance books provide a chapter or section on statistics ([Jai91], [Lil05], [SW07]). Statistics refers to a range of techniques for analyzing data, interpreting data, displaying data, and making decisions based on data. A statistic is a numerical quantity (e.g., mean) calculated from data. Statistics is a vast, well-studied science. Not all aspects of statistics are applicable to performance, so only relevant terms and concepts are discussed here. Nonetheless, the absence of something does not imply that it cannot be used in our performance world. We will divide statistics into two basic areas: descriptive statistics and predictive statistics. The former is for anyone looking at measurements; the latter is for anyone doing modeling. We will forgo the foundational discussions of populations, samples, and random variables, and take the viewpoint that you have a bunch of numbers that you have to summarize and/or understand. Of course, understanding those foundational concepts will improve your analyses.